System and method for object extraction

ABSTRACT

A system and method for capturing images from various angles of an object that is resting on an orientation indication background, using a camera of a mobile device and visible light. The images are processed to construct a three-dimensional image model and a representation of the three-dimensional image model are displayed on an electronic display screen and manipulated or embedded into a video game.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/097,950, filed Apr. 13, 2016, which application claims the benefit ofU.S. Provisional Application No. 62/275,270, filed Jan. 6, 2016, andwhich is a continuation-in-part of U.S. application Ser. No. 13/201,512,filed Aug. 15, 2011, which is the U.S. National Stage of InternationalApplication PCT/US2010/044343, filed Aug. 4, 2010, which claims thebenefit of U.S. Provisional Application No. 61/231,216, filed Aug. 4,2009, all of which are fully incorporated herein by reference for allpurposes.

FIELD OF THE INVENTION

This application relates to an interactive computer imaging system andmethod, and in particular to a system and method for identifying anobject extracted from an image.

BACKGROUND

Construction sets may include standardized interlocking pieces thatallow for the construction of a variety of different models or shapes.These pieces may not require special training or design time toconstruct complex systems. Interlocking pieces may be suitable forcreating temporary structures for use as a toy for children. One exampleof an interlocking construction set is LEGO® (LEGO Juris A/SCorporation, Denmark), which can include colorful interlocking plasticbricks and an accompanying array of gears, minifigures and various otherparts. These interlocking bricks can be assembled and connected in manyways, to construct such objects as vehicles, buildings, and even workingrobots. Anything constructed by these interlocking bricks can be takenapart, and the pieces may be used to make other objects.

SUMMARY OF THE INVENTION

In one aspect, the invention is a method of extracting images of areal-world object and displaying a representation of a three-dimensionalmodel of the object in such images in an electronic display of theobject. In this aspect, the invention comprises: capturing, using animage capture device and visible light, a plurality of two dimensionalimages of said real world object, said object at a fixed positionrelative to an orientation indicator visible in at least some of saidcaptured two-dimensional images, a first image of said plurality of saidcaptured two dimensional images captured from a first orientation ofsaid object to the image capture device capturing said plurality ofimages, and a second image of said plurality of images captured from asecond orientation of said object to said image capture device;calculating said first orientation of said object from a position ofsaid orientation indicator in said first image, and calculating saidsecond orientation of said object from a position of said orientationindicator in said second image; constructing a three-dimensional imagemodel of said object from said plurality of captured images; deletingdata of said orientation indicator prior to said construction; andembedding a representation of said three-dimensional image model into anelectronic display of said object.

In another aspect, the invention is a system for extracting images of areal-world object and displaying a representation of a three-dimensionalmodel of said object in such images in an electronic display of theobject. In this aspect, the system comprises: an image capture device, amemory; a processor; and a mat having visible thereon an orientationindicator, and configured to support said object; wherein said imagecapture device is configured to capture a plurality of images of saidobject using visible light, a first image of said plurality of imagesfrom a first orientation relative to said object, and a second image ofsaid plurality of images from a second orientation relative to saidobject; and said processor is configured to calculate said firstorientation of said object from a position of said orientation indicatorin said first image, and calculate said second orientation of saidobject from a position said orientation indicator in said second image;construct a three-dimensional image model of said object from saidplurality of captured images; delete data of said orientation indicatorprior to said construction; and embed a representation of saidthree-dimensional image model into an electronic display of said object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a schematic diagram of a system in accordance with anembodiment of the invention;

FIG. 2 depicts a schematic diagram of components of the system of FIG.1;

FIG. 3 depicts an image captured by the system depicted in FIG. 1;

FIG. 4 depicts a method in accordance with an embodiment of theinvention;

FIG. 5 depicts a representation of an image in accordance with a step ofthe method depicted in FIG. 4;

FIG. 6 depicts a representation of another step in the method depictedin FIG. 4;

FIG. 7 depicts a close up of a portion of the image of FIG. 5;

FIG. 8 depicts an alternate embodiment of an element of the image ofFIG. 3;

FIGS. 9A, 9B and 9C depict a representation of another step in themethod depicted in FIG. 4;

FIG. 10 depicts an extracted image from the image of FIG. 3 inaccordance with the method depicted in FIG. 4;

FIG. 11 depicts a method in accordance with an embodiment of theinvention;

FIG. 12 depicts a method in accordance with an embodiment of theinvention;

FIG. 13A depicts an assortment of real world objects;

FIG. 13B depicts the assortment of objects of FIG. 11A embedded in avideo game;

FIG. 14 depicts a representation of a result in accordance with themethod of FIG. 2 presented on a handheld mobile device;

FIG. 15 depicts, a schematic diagram of an object to be imaged restingon a mat in accordance with an embodiment of the invention; and

FIG. 16 is a flow diagram of a method of extracting images of areal-world object and embedding representation of such images in anelectronic display of the object, in accordance with an embodiment ofthe invention.

DETAILED DESCRIPTION

In the following description, various embodiments of the invention willbe described. For purposes of explanation, specific examples are setforth in order to provide a thorough understanding of at least oneembodiment of the invention. However, it will also be apparent to oneskilled in the art that other embodiments of the invention are notlimited to the examples described herein. Furthermore, well-knownfeatures may be omitted or simplified in order not to obscureembodiments of the invention described herein.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specification,discussions utilizing terms such as “selecting,” “evaluating,”“processing,” “computing,” “calculating,” “associating,” “determining,”“designating,” “allocating” or the like, refer to the actions and/orprocesses of a computer, computer processor or computing system, orsimilar electronic computing device, that manipulate and/or transformdata represented as physical, such as electronic, quantities within thecomputing system's registers and/or memories into other data similarlyrepresented as physical quantities within the computing system'smemories, registers or other such information storage, transmission ordisplay devices.

The processes and functions presented herein are not inherently relatedto any particular computer, network or other apparatus. Embodiments ofthe invention described herein are not described with reference to anyparticular programming language, machine code, etc. It will beappreciated that a variety of programming languages, network systems,protocols or hardware configurations may be used to implement theteachings of the embodiments of the invention as described herein. Insome embodiments, one or more methods of embodiments of the inventionmay be stored as instructions or code in an article such as a memorydevice, where such instructions upon execution by a processor orcomputer result in the execution of a method of an embodiment of theinvention.

This application relates to interactive games and computer visionimaging systems that may extract and identify an object captured in animage. The object may be created by assembling interlocking bricks(e.g., LEGO® bricks or building units) or other pieces or buildingblocks. In one embodiment, the bricks or construction objects used haveor conform to a known set of dimensions (e.g., a known and limited setof shapes and dimensions). The image may be captured in uncontrolledlighting conditions. The identification of the object may be based onshape analysis and/or shape comparison. Identification can be done byrecognizing and classifying the object by comparison to a record in apredefined database of objects. By embedding the extracted image of thephysical object within an interactive video game, a user may experiencea level of interaction with the extracted object.

FIG. 1 is a schematic diagram of system 100 in accordance with anembodiment of the invention. System 100 may include computer such as aworkstation or personal computer 110 and digital imaging device 190.Personal computer 110 may include a processor, a display 292 (FIG. 2), auser interface(s) input/output device 296 (FIG. 2) (e.g., keyboard,arrow keys, and/or mouse), and memory 245 (FIG. 2). Digital imagingdevice 190 may be, by way of example, an imager, a webcam or a digitalcamera. Connected to personal computer 110 may be datastore 170containing a database of images, and other data (e.g., software orcode). Datastore 170 may be implemented by a memory or another storagedevice, such as a hard disk drive.

System 100 may be a dedicated, stand-alone device having a processor, adisplay, a user interface, memory, database, and a digital imagingdevice. System 100 may be embodied in other computing devices, such asnotebook or netbook 120, personal digital assistant (PDA) 130, mobilephone 140, or tablet (pad) computer 150. System 100 may include anintegral imaging device in, for example, PDA 130 and mobile phone 140. Adigital imaging device 190, 192, 194 may be connected, respectively, topersonal computer 110, notebook or netbook 120, and tablet computer 150.

System 100 may include a computer program application stored innon-volatile memory, or computer-readable medium (e.g., hard drive,flash memory, CD ROM, magnetic media, etc.). The computer programapplication may include code or executable instructions that whenexecuted may instruct or cause the controller or processor of system 100to perform methods discussed herein such as a method of extracting,identifying, or embedding the object.

In another embodiment, computing device 110, 120, 130, 140, 150 ofsystem 100 may each be configured as client devices connected throughelectronic communication network 160 to a remote server 180. Electroniccommunication network 160 may be the Internet, a local area network, awide area network, or other suitable configurations of an electroniccommunication network. Client device computing devices 110, 120, 130,140, 150 may have a local client application, and remote server 180 mayinclude a remote server application. In combination, the clientapplication and remote server application may provide the instructionsfor system 100 to perform methods discussed herein such as a method ofextracting, identifying, and/or embedding the object. A datastore 172containing a database of images may be connected to remote server 180.In one embodiment, stand-alone datastore 174 containing a database ofimages may be connected to electronic communication network 160.Datastore 174 may be accessed through electronic communication network160 by computing devices 110, 120, 130, 140, 150 and/or remote server180.

System 100 may provide an interactive system that can detect and extractan image of a physical object from an image of a real world scenecaptured by digital imaging device 190, 192, 194. System 100 may modelthe extracted object on a computer display for visualization andembedding in a computer video game. The object may be constructed frominterlocking bricks, or other items, or other materials. In oneembodiment, system 100 may provide instructions to a user on a suggestedshape or configuration for the object to be constructed—for example, aplane, an automobile, a house, a character, etc. For example, aprocessor or controller may select a shape and present the shape on adisplay possibly along with an instruction to a user to construct thedisplayed shape from the bricks, so as to create the physical object.The suggested configurations can be stored in, for example, datastores170, 172, 174. After a suggested object is extracted, system 100 maycompare the extracted image to the suggested configuration and maycompute a rating or metric that is representative of the comparisonresults, or of the correspondence between the image or the physicalobject to the suggested configuration or shape.

FIG. 2 depicts a schematic of components of system 100. System 100 mayinclude controller or central processor unit 270 that may be connectedto an internal bus 230. Digital imaging device 190 may be connected toCPU 270 via an input/output port (not shown). Also connected to CPU 270via an input/output port may be memory or datastore 170. In an alternateembodiment, the shape configurations stored within the datastore may bestored in memory 245 coupled to internal bus 230, thus, reducing theneed for datastore 170.

CPU 270 may provide background detector 240 with a digital imageprovided by the digital imaging device. Background detector 240 may beimplemented by dedicated hardware, software modules, and/or firmware,where CPU 270 executes the instructions. Other units discussed hereinmay also be implemented by dedicated hardware units, software modules,and/or firmware, where CPU 270 executes the instructions. The boundariesand edges of background 220 may be calculated and extracted bybackground detector 240 using an edge detector process or algorithm. Todiscriminate background 220 (FIG. 3) from image 205 (FIG. 3) (e.g., adigital image) of a physical object, e.g., made of bricks or otherbuilding units, the curvature of the edges may be calculated andlocations where the curvature is about 90 degrees may be marked ascorners. Each detected corner may be associated with the curvature ofedges that are connected to it. The background detector may provide itsresult to CPU 270 for storage in internal memory 245, or an externalmemory unit, for example as a record in datastore 170. Background 220may include a grid, pattern, or visual cues to enable extraction of theimage of the physical object. A background need not be used.

Perspective transformation correction (PTC) unit 250 may compute acorrected virtual grid substantially aligned to the image of the object210. Color calibration extractor unit 260 may use the perspectivecorrected grid to locate color calibration palette(s) formed onbackground 220. The color value of the associated pixel from image 205corresponding to the bricks of the palette may be extracted andconverted to another color space representation by the color calibrationextractor unit. Also, a few calibration points from the background fieldof background 220, chosen to represent the field color, may also beextracted and converted to the same HSV (hue, saturation, and value)color space by color calibration extractor unit 260. Other color spacesmay be used, and color need not be used.

Brick identifier unit 280 may sample and extract the value of associatedpixels of image 205 in a few different places. These values may beconverted to for example the HSV color space. The hue value of theconverted data may be compared with the hue values of the colorcalibration palette, and optionally the field of background 220. Thecolor with the smallest difference is chosen by brick identifier unit280 to represent the color of this grid location. Brick identifier unit280 compares the intensity and saturation levels of the pixels todetermine if the brick is colored, black or white. If the determinedcolor of the brick is not the color of the background, then brickidentifier unit 280 discerns that this brick is part of the constructedobject.

Comparison unit 290 may compare the shape of the detected image with arecord stored in a database. The comparison is done by comparison unitusing for example a correlation function.

FIG. 3 depicts image 205 of object 210 placed on background 220. Object210 may be a real world, physical object made of building bricks orother units. Similarly, background 220 may be a real-world, physicalobject made of building bricks or other units, or printed on paper orcardboard. A background need not be used.

The object to be extracted can be captured on a background having apredefined pattern. This predefined pattern may have predetermined knownspatial features that can discriminate the pixels belonging to object210 from the pixels belonging to background 220. For example, FIG. 3depicts background 220 having a square grid. The spatial pattern ofbackground 220 may be discriminated from pixels of image 205 containingobject 210 made from interlocking building bricks, which are typicallysolid and rigid. The predefined grid need not be a square grid, andother implementations of a predefined background pattern may be used.

With a result similar to the chroma key technique, where foregroundobjects are separated from a background using a bluescreen orgreenscreen, system 100 may extract object 210 from image 205 by usingthe extracted object's spatial and morphological features, and may bedone independent of color recognition.

The captured image containing object 210 may be analyzed to extract theobject from background 220. Once extracted, object 210 may be analyzedfor shape identification by comparison to a predefined database ofobjects. Once the shape of object 210 is identified, the digitalextracted object can be used in a variety of dynamic interactions with aplayer.

FIG. 4 depicts process 400 in accordance with an embodiment of theinvention. A mask of object 210 may be extracted using a morphologicaloperator such as a 2D bottom hat operator, which may give an image thatcan then be filtered using a combination of a threshold techniques andother morphological operators (e.g., closing and opening) to removespurious artifacts. The resultant mask created by applying thesemorphological operators may represent object 210 detected and isolatedfrom background 220.

In one embodiment, process 400 for extracting and identifying object 210may be performed as follows:

Image 205 may be obtained, step 410, by arranging object 210 onbackground 220, and taking a digital image of the arrangement. The imagemay be taken using digital imaging device 190 connected to computingdevice 110, a personal computer. Alternatively, the image may be takenusing other computing devices, described above, and either external orinternal digital imaging devices associated with these other computingdevices.

Background Object Detection:

The background object on which the bricks may be placed can be anysurface with known features such as a specific color, a specific spatialpattern, or other spectral/spatial feature(s) that may aid in thedetection and extraction of the object. Background object 220 may be,for example, a printed paper or cardboard, or a surface formed frominterlocking bricks. If for example, an interlocking brick background isused as a background, process 400 may include detecting four highcontrast corners, which can be created by using, for example, whiteperimeter bricks encompassing a dark color field. The corners can beused to find the background field's boundaries in the following way:

The boundaries of the image may be calculated and extracted using anyedge detector which can detect edges in images, for example, the Cannyalgorithm is one such multi-stage algorithm. The Canny algorithm may usean edge detection operator that may include four filters to detecthorizontal, vertical and diagonal edges in an image. The edge detectionoperator may return a value for the first derivative in the horizontaldirection and the vertical direction. From these derivatives an edgegradient and direction can be determined. Other non-maximal suppressionedge detector techniques may also be used by process 400.

To discriminate background 220 from image 205, the curvature of theedges may be calculated and locations where the curvature is about 90degrees may be marked as corners, step 420. FIG. 5 depicts aninterlocking brick background and object with corners identified byprocess 400 and indicated by a “+” mark in the figure. Each of themarked corners may be considered a candidate corner of the backgroundfield.

With reference to FIG. 5, each detected candidate corner may beassociated with the curvature of edges that are connected to it. Theseedge curvatures may be on a straight line if they are found, within apredetermined threshold, to be close to zero. In such a case, thecorners may be marked as candidates for being the background field'scorners. The slope of each associated edge of the candidate corners iscalculated and its line equation is extracted. If the distance between afirst candidate corner and one of the line equations associated with asecond candidate corner is smaller than some predetermined threshold,then those two corners are accounted as being on the same line. If foursuch corners are found where the lines form a quadrangle the backgroundfield's corners are assumed to be found. A candidate corner that is noton the same line with other corners may be removed from consideration asa background corner.

Perspective Transformation Correction:

Background 220 may be rectangular or square, or other shapes, but due tothe fact that the background may not necessarily be orientatedperpendicular to the camera lens' axis, the acquired image may not besquare and true. The position of the detected corners of background 220may be used to perform a perspective transformation calculation, step430. This perspective transformation may be used to compute a correctedvirtual grid that is substantially aligned with the image of theinterlocking bricks forming object 210 as depicted in image 205. FIG. 6depicts a representation of image 205 with a superimposed true gridafter the perspective transformation calculation.

Calibration Color Extraction:

FIG. 7 depicts a close up of one corner of background 220 depicting acolor calibration palette 610. Background 220 may have color calibrationpalette 610 located in one or more of its four corners. Each colorcalibration palette may include, for example, four different coloredblocks—e.g., red, yellow, green, and blue. Other numbers of colors andother colors may be used. The colored blocks may be formed, for example,using interlocking bricks when background 220 is assembled.

Using the perspective corrected grid obtained in step 430, the locationof each of these colored bricks forming color calibration palette 610may be found. The color value of the associated pixel from image 205corresponding to the bricks of the palette may be extracted, step 440,and converted to another color space representation, such as HSV colorspace. Also, a few calibration points from the background field ofbackground 220, chosen to represent the field color, may also beextracted and converted to the same HSV color space.

Colored building blocks can undergo additional analysis, andclassification of their colors to obtain a color model of the object canbe made.

FIG. 8 depicts an alternate embodiment of background 220. In thisembodiment, background 220 may be printed on a suitable flat surface andmaterial, such as paper or cardboard. Here, background 220 may includecolor calibration palette 620 which may include a combination of colorsarranged in a specific spatial arrangement. Color calibration palette620 may allow for the calibration of colors to compensate for whitebalance, dynamic range and color consistency.

Brick Identification:

For each grid location, the value of the associated pixels of image 205may be sampled in a few different places and extracted, step 450. Thesevalues may be converted to HSV color space. The hue value of theconverted data may be compared with the hue values of the colorcalibration palette, and optionally the background field, obtainedduring step 440. The color with the smallest difference may be chosen torepresent the color of this grid location. The intensity level of theassociated pixels may be compared with the intensity level and thesaturation level of the black and white pixels associated with thebricks used in the four corners of background 220. If the saturationlevel is closer to the saturation level of the white color than to thesaturation level of the color palette and the background; and if itsintensity level is also high, then the color of the grid location may bemarked as white. If the intensity level is very low and close to theintensity level of the black color, then the grid location may be markedas black. Otherwise, the color of the grid location is assigned thecorresponding color from the color calibration palette.

FIGS. 9A-C depicts interim results of step 440. A morphological operatorsuch as a 2D bottom hat operator will give an image (see FIG. 9A) whichcan then be filtered using one, or more, threshold techniques (see FIG.9B) and other morphological operators—e.g., closing and opening, toremove spurious artifacts. The resultant mask created by applying thesemorphological operators may represent object 210 isolated frombackground 220 (see FIG. 9C). FIG. 10 depicts an example of an image ofobject 210 after extraction from image 205 by process 400.

After the image of object 210 is separated from background 220 andextracted from image 205, process 400 may perform further analysis ofthe orientation of the extracted image by applying binary large object(BLOB) analysis, labeling, and moments calculations to obtain acalculated angle that can be used to rotate the object to a more desiredangle.

In one embodiment, further analysis of the spatial and spectral featuresassociated with the detected object can be used to train system 100 torecognize the object in the future by, for example, examining the colorhistogram of the detected object and relations of colored segmentswithin the detected object. This information may be stored in datastore170, 172, and 174.

Shape Comparison and Scoring:

The shape of the extracted object from image 205 may be compared, steps460, 470 with a record of a database stored within datastore 170, 172 or174. The extracted object shape can be compared to the input shape fromthe database by using, for example, a normalized correlation function,cor:

f(cor)=(stored configuration, extracted shape)

f(cor)=1, if stored configuration=extracted shape

where:the stored configuration may be within a datastore, and the extractedshape may be provided, for example, by process 400.

Other operations can be used. In one embodiment, a game can be createdbased on the extracted image and its classification. The computer mayengage the user in an interactive manner to create real world objectswhich are than provided to the computer in an image for extraction. Acomputer may present a record from a database on a computer display andask a player to build the displayed object. The displayed object may be,for example, a car. After the object is assembled by the player, animage of the object placed on a suitable background (as discussed above)can be obtained. The computer may than extract an image of the object,and compare it to the database record. FIG. 13A depicts a variety ofobjects that can be built. By way of example, FIG. 13A depicts realworld objects in the form of character 1010, car 1020, tree 1030,airplane 1040, face 1050, and house 1060. The following text is anexample of an interactive script which may be outputted to a user andthe corresponding actions in accordance with this embodiment:

-   -   Computer: Welcome, players, let's go on a fun storytelling        adventure!    -   Computer: First, we need a character . . . . Can you build a boy        character?    -   [Player builds and shows a body of a character on the        background]    -   Computer [after extracting and comparing the object]: Cool, now        let's give him a face; can you build a face for him?    -   [Player builds and shows a face on the background]    -   Computer: Fantastic . . . mmm . . . . Let's build him a house .        . . can you help do that?    -   [Player builds and shows a house on the background]    -   Computer: That's a really nice house.    -   Computer: Hey, what's that sound? Is that an airplane? Can you        build an airplane?    -   [Player builds and shows an airplane on the background]    -   Computer: And what is this sound? Is that a car honking? Let's        build one . . . .    -   [Player builds and shows a car on the background]    -   If showing the wrong thing than computer responds with:    -   Computer: That's a cool [face/house/airplane/car/tree] but let's        try again.

By obtaining an image of the constructed object, the computing devicemay detect the presence of a shape and extract the shape showing adigital representation of the extracted shape on a display screen. Theuser can create a virtual world by manipulating the location or otheraspects of the digital representation of the shape on the screen.

The extraction can also involve automatic recognition of the shape byusing image processing techniques. Using automatic recognition, thecomputer can ask the user to build a specific shape and may give somefeedback based on a comparison with a predefined image in the database.

FIG. 11 depicts process 700 which may implement a game, for example,including interactive script presented above. A computer instructs(e.g., via a display or monitor) a player, step 710, to construct anobject. The instruction can be provided, for example, by audio (verbalor sound) through a sound card, or visually by presenting written wordson a computer display. An image of the object may or may not bedisplayed. By not displaying an image of the instructed object, theplayer's knowledge, word skill, and imagination may be exercised. Theobject may be chosen at random from a database, or may be chosen in apredetermined sequence. The predetermined sequence may be selected sothat the player's construction skills are developed by the sequence ofobject selection.

After the player constructs the object, the player may place the objecton a suitable background, as describe above, and take a digital image ofthe object and background using a digital imaging device. A backgroundneed not be used. The computer obtains, step 720, the digital image ofthe physical object as an image file from the digital imaging device.

The image of the constructed physical object may be extracted, step 730,from the digital image. The extraction may be performed using, forexample, the method of process 400 described above with reference toFIG. 4.

The object's extracted image may be compared to a database record thatcorresponds to the instructed object, step 740. In one embodiment of theinvention, the comparison may provide a rating or metric that may beindicative of the constructed object's conformance with the instructedobject.

The extracted image may be embedded, step 750, in a video game andutilized within the game parameters by the player.

Other steps or series of steps may be used.

In another embodiment, the computer may randomly choose a shape (e.g., apredetermined shape taken from a memory or a database) and show ordisplay the shape on the display device. A player may try and constructthat shape using bricks or other construction objects, for example,under timed conditions, as part of a game. Once the player finishesconstructing the shape, he uses the digital imager to upload an image ofhis creation to the computer. After the computer extracts and identifiesthe object, it can be compared to the original shape shown on thedevice. The result of comparison can be shown to indicate a success orfailure in the construction task as part of an interactive competition.

A user may be presented with a selected shape composed from bricks orother building units appearing on the screen of a computing device. Thisselected shape may also be accompanied by a timer that starts when theuser first sees the shape. The timer may be used to measure how fast theuser can successfully build the shape using the interlocking bricks.

The user tries to build the shape using regular bricks and may place theobject he built on the area designated as the background area, if abackground is used. The computing device may constantly analyze imagesacquired from a digital camera and may detect the presence of an objecton the background area.

If an object is detected, an embodiment of the method may extract theshape of the object the user created by using processes in embodimentsof the present invention, for example, process 400 described above.

The process may compare the extracted image of the object built by theuser to the selected shape that was presented as the target shape todetermine how successful the user was in building the shape. Thecomparison may provide a rating or metric based on how accurate theuser's object corresponds to the selected shape. This rating or metricmay also include a component indicative of the time spent to constructthe object, where the component varies based on the complexity of theobject.

FIG. 12 depicts process 800 which may implement the timed constructiondescribed above. A computer may retrieve a record, step 810, from adatabase containing records representing predetermined shapes. A visualimage of the retrieved shape may be displayed, step 820, on a computerdisplay.

The computer may provide a message requesting, step 830, the player toconstruct the object seen on the display. Optionally, the computer mayalso present a clock timer on the screen. The timer (whether displayedor not) may be a count up, or a count down timer so as to track the timeelapsed for the user to construct the physical object.

After receiving an indication (e.g., via a user input device) that theuser has completed construction, or waiting for a predetermined timeperiod to elapse, step 840, the computer instructs the player to placethe object on a suitable background, as describe above, and take adigital image of the object and background using a digital imagingdevice. A background need not be used. The computer may obtain, step850, the digital image as an image file from the digital imaging device.

The image of the constructed object may be extracted, step 860, from thedigital image. The extraction may be performed using, for example, themethod of process 400 described above with reference to FIG. 4.

The object's extracted image may be compared to the retrieved databaserecord, step 870. The comparison may provide a rating or metric that maybe indicative of the constructed object's conformance with the retrievedrecord.

The extracted image may be embedded, step 880, in a video game andutilized within the game parameters by the player.

Other steps or series of steps may be used.

The extracted image (or multiple extracted images of various physicalobjects) can be digitally represented on the display device as part of avirtual world, or video game, where the objects inhibiting the virtualworld and/or video game, were designed and built from the constructionset in the real-world. FIG. 13B depicts the extracted images of themultiple real world objects of FIG. 13A embedded in a computer graphic.

FIG. 14 depicts extracted image 110, after being processed from an imageof real world combination 1020 containing an interlocking brickbackground and object. FIG. 14 depicts extracted image on mobile device130, 140 equipped with an internal camera.

Reference is made to FIG. 15, a schematic diagram of an object to beimaged resting on and supported by a mat in accordance with anembodiment of the invention. In some embodiments, an object 1500 such asa toy or model may be placed upon a mat 1502 or other surface. Mat 1502may include a visible pattern 1504 that may appear in an image capturedof the mat 1502 and the object 1500 and that may indicate in such imagean orientation of the mat 1502 and the object 1500 resting on the mat1502. In some embodiments, mat 1502 may be placed upon a wheel or otherrotatable surface so that mat 1502 and object 1500 may be rotated inview of a camera of a portable device while images of the mat 1502 andobject 1500 are captured from a reasonably steady platform and uniformheight and distance.

In some embodiments, object 1500 may be or include a construction blockbut may be other than a construction block. For example, object 1500 maybe or include a figure, action figure, toy, doll, door knob or anyobject made from any material such as plastic, paper, clay or metal. Insome embodiments, object 1500 to be imaged and extracted may be placedon or against mat 1500 that may have pattern 1504 printed thereon sothat one or more images of the object 1500 against the mat 1502 mayindicate an orientation from which the image was captured. Images of theobject 1500 placed or resting on the background mat 1502 may be capturedfrom various angles and processed, by known methods such as for example:

Control points recognition—Using an algorithm for detecting corners orother control points such as Harris corner detector or keypointsrecognition to find corners or control points in the images of theobject 1500.

Recognition of significant control points such as main corners byrecognizing two, three or more control points of a high contrast framein the printed pattern.

Determining a perspective transformation and rectifying the image byusing a 2D to 2D transformation, also known as homography that maps thecaptured pattern to rectify coordinates.

Verification of the pattern by checking for example colored areas orother a priori known patterns, and/or by checking specific pointslocated relative to a high contrast frame.

Finding an orientation and position of the pattern on the mat orbackground based on specific known features located in one or more areasof the pattern: Once an orientation is derived and a position of a firstcorner is understood, there may be defined for example a closest cornerto the specific known feature of the object 1500. Other features maythen be plotted in coordinates of for example x, y, z=0,0,0 in a worldcoordinate system. Coordinates of one or more control points may bedetermined, and imposed on coordinates of the captured images (screencoordinates).

Using the relations between corner positions in the world coordinatesystem and screen coordinates, a camera matrix (one or more of intrinsicand extrinsic) may be extracted. This may be performed for example usingSVD (for first guess) and square error minimization algorithms. A cameramatrix may provide a position and rotation of the camera relative to theworld coordinates.

To separate the scanned object from the printed background, a backgroundsubtraction process may be used based on color features (like a greenscreen effect) or spatial features or both may be used. If the patternitself is extracted with the silhouette it may be removed in the voxelcarving sequence.

Voxel carving may be performed on for example some or all of the voxelsin the region of interest (which may be any cubic D×D×D millimeters).For each voxel subject to the carving, determining its 8 vertices andback projecting their 3D coordinates into the screen coordinates isperformed. A determination may be made as to whether or not those screencoordinates are inside the silhouette. If all 8 vertices of a voxel areoutside of the silhouette, then this voxel may be “carved” or “turnedoff”.

The image capture device may be rotated around the object 1500 or themat 1502 may be rotated with the object resting thereon and still orvideo images may be captured. A full digital representation of theobject may thereby be created.

In some embodiments one or more of the following processes may beperformed:

Color information may be gathered from the image about vertices of thecontour voxels in one or more of the images in which such voxels arevisible. A color of the vertices may be determined from a most saturatedpixel, or median color or average color or any other relevant methods.

In some embodiments a smoothing of the mesh may be implemented using asmoothing algorithm such as a marching cubes algorithm.

Texturing of triangles may be achieved by gathering color data of a backprojection of one or triangles that face the camera to create a textureto the mesh.

Stereoscopic 3D reconstruction may be performed by creating depth mapsfrom couples of captured images to further carve the voxels and havemodels with cavities being determined.

Objects may be classified as being for example a model type, such ashumanoid, four-legged animal, car, etc. A smart rig or skeleton for themodel type may be created or designated and a determination may be madeas to which vertices of the imaged models are to be attributed to one ormore skeleton part. Digitally animating the model may be performed withpredefined animations such as walking, jumping, flying etc.

Image models may be presented in for example a CAD environment thatallows a user to show, rotate, move and zoom in/out of the extractedimage of the object. Image models may be presented in an augmentedreality format (with or without animations) such as over the mat, orover other models/patterns. Models may be presented in a virtual realityenvironment: showing the model in 3D by rendering the scene from atleast two angles and showing the images side by side on a same screen soit will be compatible with for example Google cardboard or Samsung gearVR.

In some embodiments an extracted and digital representation of a scannedobject whether smoothed or not, may be transmitted to for example a 3Dprinter to create a physical copy of the object 1500.

Some or all of the images of the object may be captured using a 2Dcamera as may be included in a mobile device 1520 such as a smarttelephone, laptop or table computer that may include a processor 1522and a memory 1524. The mobile device 1520 may for example be movedaround the object 1500 to be imaged, or the mobile device 1520 may beheld or placed in a stand and the object 1500 on the mat 1502 may beturned as the images are captured. Processing of the images into themodeled object may also be performed in real time using processor 1522or some other processor. In some embodiments, camera calibration may beperformed during the image capture process. The pattern 1504 on the mat1502 or background may be removed simultaneously or in real time withthe capturing of the images or at any time prior to the construction ofthe 3D representation of the captured images. Image capture and scanningof the object 1500 may be performed in free hand scanning with freedomof the angles from which the images are captured in all axis. Suchscanning and image capture may allow images to be captured by forexample a hand-held device rather than requiring that the imager bemounted in a fixed height and orientation. In some embodiments carvingof the voxels may be performed in real time while the images of theobject are being captured by the imager associated with the mobiledevice. In some embodiments mat 1502 may be constructed from a highlyretroreflective material and the imager used to capture images of theobject 1500 will use a lighting source such as an LED to create an imagewhere the object 1500 may be easily extracted or differentiated from mat1502.

Reference is made to FIG. 16, a flow diagram of a method of extractingimages of a real-world object and embedding for example 3D or otherrepresentation of such images in an electronic display of the object, inaccordance with an embodiment of the invention.

In block 1600, a mobile imager in for example a housing of a mobiledevice may capture images of a real world object using visible light.The object may be resting on or in front of and at a fixed positionrelative to an orientation indicator that appears in at least some ofthe captured images. A first of the images may be captured from a firstorientation or angle of the object to camera, and a second of the imagesmay be captured from a second orientation or angle of the object to thecamera. Other numbers of images may captured, such that for exampleimages of the object from many sides or orientations may be collectedand subject to the processes below.

In block 1602 an orientation or position of the camera during thecapture of the first image relative to one or more of the object and theorientation indicator may be calculated. The orientation of the cameraduring the capture of the second image relative to one or more of theobject and the orientation indicator may be also be calculated.

In block 1604, a processor may develop or construct a three-dimensionalimage model of the object from the captured images.

In block 1606, data about the orientation indicator as may have appearedin the captured images may be deleted from the process of constructingthe three-dimensional image model.

In block 1608, data from the three-dimensional image model may beembedded into a representation the three-dimensional image model andsuch representation may be displayed on an electronic display.

In some embodiments a real world copy of the object may be constructedby a three-dimensional printer, using data from the three-dimensionalimage model.

In some embodiments, the camera and its housing may be rotated aroundthe real world object during the capturing of the images.

In some embodiments, the real world object may be rotated along with themat while the camera is held still during the image capturing.

In some embodiments, the constructing of the three-dimensional imagemodel is performed by the processor in the housing of the device thatincludes the imager.

In some embodiments deleting image data of the orientation indicator aswas captured in the first image is performed prior to capturing of thesecond image.

Some embodiments may include detecting one or more of the same controlpoints of the real world object that appear in a first image and in thesecond image, and determining a perspective transformation of thecontrol points between the first image and the second image.

Some embodiments may include manipulating the representation of thethree-dimensional image model on the electronic display. In someembodiments the three-dimensional image model may be embedded in view ofa video game that displayed on the electronic display.

While there have been shown and described fundamental novel features ofthe invention as applied to several embodiments, it will be understoodthat various omissions, substitutions, and changes in the form, detail,and operation of the illustrated embodiments may be made by thoseskilled in the art without departing from the spirit and scope of theinvention. Substitutions of elements from one embodiment to another arealso fully intended and contemplated. The invention is defined solelywith regard to the claims appended hereto, and equivalents of therecitations therein.

We claim:
 1. A computer-implemented method comprising: (A) capturing,with a two-dimensional (2-D) image capture device, multiple distinct 2-Dimages of an arbitrary object, said object at a fixed position on aregion having a known background pattern thereon, said multiple 2-Dimages corresponding to multiple orientations or positions of said 2-Dimage capture device relative to said object; (B) in at least some ofsaid multiple 2-D images, separating the object from the knownbackground pattern to form multiple silhouettes corresponding to saidobject; (C) obtaining a three-dimensional (3D) image model of saidobject, said obtaining using at least some of the multiple silhouettes;and then (D) embedding a digital representation of saidthree-dimensional image model into an electronic display of said object.2. The method of claim 1 wherein the capturing in (A) uses visiblelight.
 3. The method of claim 1 wherein the known background pattern isvisible.
 4. The method of claim 1 wherein, in (B), said separating usesspectral features of the object or the background pattern.
 5. The methodof claim 1 wherein, in (B), said separating uses spatial features of theobject or the background pattern.
 6. The method of claim 1 wherein, in(B), said separating uses color features of the object or the backgroundpattern.
 7. The method of claim 1 wherein the object made from materialselected from: plastic, paper, clay, and metal.
 8. The method of claim 1wherein the region having a known background pattern thereon is a mat.9. The method of claim 1 wherein the region having a known backgroundpattern thereon comprises a retroreflective material.
 10. The method ofclaim 9 wherein the 2-D image capture device uses a lighting source toperform said capturing in (A).
 11. The method of claim 10 wherein saidlighting source is an LED.
 12. The method of claim 1 wherein the knownbackground pattern comprises at least two colors.
 13. The method ofclaim 12 wherein the known background pattern comprises a grid.
 14. Themethod of claim 13 wherein the known background pattern comprises twocolor grid.
 15. The method of claim 1 wherein the known backgroundpattern comprises a visible orientation indicator.
 16. The method ofclaim 1 wherein the region having a known background pattern thereoncomprises known features including one or more of: at least one specificcolor, and at least one specific spatial pattern.
 17. The method ofclaim 1 wherein the known background pattern has known spatial features.18. The method of claim 1 wherein the region having a known backgroundpattern thereon comprises a color calibration palette.
 19. The method ofclaim 1 further comprising: color calibrating said three-dimensionalimage model using said known background pattern.
 20. The method of claim1 further comprising: calibrating said image capture device during saidcapturing.
 21. The method of claim 1 wherein obtaining the 3D imagemodel of the object in (C) comprises: performing voxel carving on saidat least some of said multiple silhouettes to obtain the 3D image modelof said object.
 22. The method of claim 21 wherein said voxel carving isperformed in (C) while 2-D images of the object are being captured in(A).
 23. The method of claim 22 wherein said voxel carving is performedin (C) in real time while 2-D images of the object are being captured.24. The method of claim 1 where said separating in (B) comprisesremoving the known background pattern from said at least some of saidmultiple 2-D images.
 25. The method of claim 1 wherein said separatingin (B) is performed simultaneously or in real time with the capturing ofthe images in (A)
 26. The method of claim 1, wherein said electronicdisplay is for a video game, and wherein the digital representation canbe manipulated by a user in said video game.
 27. The method of claim 1wherein the 2-D image capture device is in a device selected from: anotebook, a netbook, a personal digital assistant (PDA), a mobile phone,a tablet, a pad, and a computer.
 28. The method of claim 1 wherein saidmultiple 2-D images are captured in uncontrolled lighting conditions.29. The method of claim 1 wherein the known background pattern isprinted on a substrate.
 30. The method of claim 29 wherein the substrateis paper or cardboard.
 31. The method of claim 1 wherein the regionhaving the known background pattern is on a rotatable surface andwherein, during said capturing in (A), said object is rotated with saidrotatable surface to obtain multiple views of said object on saidregion.